Skip to content

CI: gem variant apply tests assert JSON codes (#1089) - #1642

Merged
Mikola Lysenko (mikolalysenko) merged 3 commits into
mainfrom
agent/fix-ci-gem-variant-json-codes
Oct 22, 2026
Merged

Mikola Lysenko (mikolalysenko) merged 3 commits into
mainfrom
agent/fix-ci-gem-variant-json-codes

Conversation

@mikolalysenko

@mikolalysenko Mikola Lysenko (mikolalysenko) commented Oct 20, 2026 •

Copy link
Copy Markdown
Collaborator

LLM Description written by Claude Code:claude-opus-5-5

Refs #1089 (the apply gem-variant slice of child 6; the tracking issue stays open)

Root cause

apply/cli_gem_variant_mismatch_policy.rs checked gem release-variant
behaviour by matching 8 English sentences on human stderr. A wording
change could fail a test whose behaviour was still right. The rest of
the apply suite already asserts --json codes in its behaviour
tests and keeps the wording in apply_render.rs.

Change

  • cli_gem_variant_mismatch_policy.rs: all 6 tests now run with
    --json and assert the exit code, status, event actions and
    errorCodes, plus the bytes on disk:
    • default overwrite: applied plus a content_mismatch_overwritten
      event for the purl.
    • --strict: failed/apply_failed for the purl, naming the
      file, and no overwrite event.
    • --force: applied plus the overwrite event. Before this, only
      the exit code and bytes were checked.
    • wrong-platform qualified singleton, and a multi-variant base where
      no variant matches: partialFailure with no applied, failed
      or overwrite event, and the file untouched.
    • mismatched sibling: only the installed variant is applied, with
      no overwrite event.
  • apply_render.rs: 5 new render tests hold the human wording:
    the mismatch warning line (checked for the purl on that line), the
    --strict hash refusal, "no matching variant found" for each of the
    two no-match cases, and no overwrite warning for a skipped sibling.
  • apply_fixture.rs: now holds the gem variant fixtures
    (run_gem_apply with BUNDLE_* scrubbed, and the singleton,
    wrong-platform and multi-variant fixtures). They reuse its existing
    install_gem, patch_record, write_manifest and stage_blob,
    which drops the policy file's private copies.

Assertions of 4+-word sentences in cli_gem_variant_mismatch_policy.rs
went from 8 to 0. No production code changes.

Testing

  • cargo test -p socket-patch-cli --all-features --test apply: 153
    passed, 0 failed, 1 ignored. Main has 148: the 6 policy tests stay,
    and 5 render tests were added.
  • cargo fmt --all -- --check: clean.
  • cargo clippy -p socket-patch-cli --all-features --test apply -- -D warnings
    reports nothing in the three changed files. Its other findings are
    already on main, in tests/common. CI's clippy command doesn't build
    tests, so it's unaffected.
  • The expected JSON shapes come from running each fixture with
    --json on main's binary.
  • CI on a641164 is green: 36 passed, 50 skipped (full-tier jobs), no failures. Bugbot found no issues.
  • No cfg(windows)/cfg(unix) code changed, so I didn't dispatch a
    full-scope run.

CI cost

Test-only change in an existing test binary. A lean PR run is the same
before and after: about 28 Linux jobs, about 8 minutes.

/code-review high findings

  • Fixed: the fixtures moved into apply_fixture.rs instead of being
    imported from a sibling test file. The no-match tests now check for no
    failed event. The --strict event must name the file. --force
    now checks the envelope. The render purl check is on the warning line
    itself. The two no-match render cases are now separate tests. No
    constants are exported without a user.
  • Not fixed here (product JSON contract, outside this test lane): in
    --json, a base with no matching release variant exits 1 with
    status: partialFailure and summary.failed: 0. Its variants show
    only as skipped/package_not_installed, and no event names the
    base or a no_matching_variant code (commands/apply.rs around
    lines 2466-2479 and 3040). A JSON consumer can't tell which package
    is still unpatched. The tests pin today's shape, so a fix that adds a
    failed event will update them deliberately.

Generated by Claude Code

Assisted-by: Claude Code:claude-opus-5-5
The gem variant mismatch tests matched English sentences on human
stderr, so a wording change could fail a test whose behaviour was
still right. They now check the --json envelope (exit code, status,
event actions and errorCodes) and the bytes on disk.

The human wording checks move to apply_render.rs, which reuses the
policy file's fixtures. The wrong-platform fixture is now a function
so both files can share it.

Refs #1089

Assisted-by: Claude Code:claude-opus-5-5
Move the gem variant fixtures into apply_fixture.rs, the shared
fixture module for the apply suite, instead of having apply_render.rs
import a sibling test file. They now reuse its gem installer and
manifest writers.

The JSON tests also check that a no-matching-variant miss reports no
failed event, that the --strict refusal names the mismatched file,
and the --force envelope. The render test checks the purl on the
warning line itself, and the two no-matching-variant cases are now
separate tests.

Refs #1089

Assisted-by: Claude Code:claude-opus-5-5
@mikolalysenko
Mikola Lysenko (mikolalysenko) marked this pull request as ready for review October 20, 2026 19:13
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

BugBot review


Generated by Claude Code

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Bugbot reviewed your changes and found no new issues!

Comment @cursor review or bugbot run to trigger another review on this PR

Reviewed by Cursor Bugbot for commit a641164. Configure here.

@mikolalysenko Mikola Lysenko (mikolalysenko) added the Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review label Oct 20, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Labeled Ready for review (burn-down agent).

  • Head: a641164. Mergeable, no conflicts; CHANGELOG.md not touched.
  • CI: 86/86 checks green (36 success, 50 skipped). No main-wide failures.
  • Bugbot: reviewed a641164, no findings; no open review threads.
  • Slack: not announced this run (no Slack send tool available); the next run will retry.

Generated by Claude Code

@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

Final review brief

What it does. The six gem release-variant mismatch tests in cli_gem_variant_mismatch_policy.rs now check the --json envelope (exit code, status, event action/purl pairs, errorCodes like content_mismatch_overwritten and apply_failed) plus the bytes on disk, instead of matching English stderr. The English wording checks move to five new tests in apply_render.rs, and the fixtures move into the shared apply_fixture.rs. Test-only; part of #1089.

Risk: low. Only test files change; no src/ change, tests/common and CHANGELOG.md untouched. All 6 old test names still exist and every old stderr check reappears in a render test; a few checks are tighter (the overwrite warning must be a skipped event for the rack purl).

Look here

  • cli_gem_variant_mismatch_policy.rs:87:`` --strict expects a `failed` event with `apply_failed` naming `lib/rack.rb`
  • cli_gem_variant_mismatch_policy.rs:155``` and :229:```` the no-matching-variant cases, asserted by absence (`partialFailure`, no `applied`/`failed` event), because the CLI records no JSON event for that miss
  • apply_fixture.rs:248-370:````` shared runner and fixtures
  • apply_render.rs:405-485:`````` the five human-wording tests

Verified

  • cargo test -p socket-patch-cli --test apply -- cli_gem_variant_mismatch_policy apply_render apply_failure_paths: 41 passed, 0 failed (none ignored; gem installs are synthesized, no ruby needed).
  • cargo fmt --check clean; no clippy warnings in the three changed files.
  • Checked that the asserted codes come from the CLI: content_mismatch_overwritten is recorded as a Skipped event in src/commands/apply.rs.
  • CI 86/86 green on a641164 (36 success, 50 skipped). Bugbot: success. 0 open review threads. Mergeable.

Changes I made: none.

Open questions (non-blocking)

  • The no-matching-variant miss has no --json event at all, so those two tests can only assert by absence. A failed/no_matching_variant event would be a reasonable product follow-up.
  • Commit 2b86b4d is empty; the squash merge drops it.

Auto-merge (squash) is armed, so approving sends this straight to the merge queue.


Generated by Claude Code

@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 22, 2026
@mikolalysenko

Copy link
Copy Markdown
Collaborator Author

[agent] Dequeued by merge-queue run 39835618551. The only failed job, e2e (ubuntu-latest, e2e_vex_build, pip:: --ignored, 22 23 24 25 26), failed because the Depot runner lost contact with GitHub after 17 min. The job has no failed step, and its annotation reads "The self-hosted runner lost communication with the server". This PR only changes apply test files, which that job doesn't run. The PR was green on a641164 and is still mergeable with no conflicts, so I re-queued it without changes.


Generated by Claude Code

Merged via the queue into main with commit eb2bfca Oct 22, 2026
86 checks passed
@mikolalysenko
Mikola Lysenko (mikolalysenko) deleted the agent/fix-ci-gem-variant-json-codes branch October 22, 2026 19:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ready for review Agent-verified: mergeable, CI green, Bugbot clean — awaiting human review

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants